Tied and Reduced RNN-T

ABSTRACT

A RNN-T model includes a prediction network configured to, at each of a plurality of times steps subsequent to an initial time step, receive a sequence of non-blank symbols. For each non-blank symbol the prediction network is also configured to generate, using a shared embedding matrix, an embedding of the corresponding non-blank symbol, assign a respective position vector to the corresponding non-blank symbol, and weight the embedding proportional to a similarity between the embedding and the respective position vector. The prediction network is also configured to generate a single embedding vector at the corresponding time step. The RNN-T model also includes a joint network configured to, at each of the plurality of time steps subsequent to the initial time step, receive the single embedding vector generated as output from the prediction network at the corresponding time step and generate a probability distribution over possible speech recognition hypotheses.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. Patent Application claims priority under 35 U.S.C. § 119(e) toU.S. Provisional Application 63/165,030, filed on Mar. 23, 2021. Thedisclosure of this prior application is considered part of thedisclosure of this application and is hereby incorporated by referencein its entirety.

TECHNICAL FIELD

This disclosure relates to a tied and reduced recurrent neuralnetwork-transducer (RNN-T) model.

BACKGROUND

Modern automated speech recognition (ASR) systems focus on providing notonly high quality (e.g., a low word error rate (WER)), but also lowlatency (e.g., a short delay between the user speaking and atranscription appearing). Moreover, when using an ASR system today thereis a demand that the ASR system decode utterances in a streaming fashionthat corresponds to real-time or even faster than real-time. Toillustrate, when an ASR system is deployed on a mobile phone thatexperiences direct user interactivity, an application on the mobilephone using the ASR system may require the speech recognition to bestreaming such that words appear on the screen as soon as they arespoken. Here, it is also likely that the user of the mobile phone has alow tolerance for latency. Due to this low tolerance, the speechrecognition strives to run on the mobile device in a manner thatminimizes an impact from latency and inaccuracy that may detrimentallyaffect the user's experience.

SUMMARY

One aspect of the disclosure provides a recurrent neuralnetwork-transducer (RNN-T) model that includes a prediction networkconfigured to, at each of a plurality of time steps subsequent to aninitial time step, receive, as input, a sequence of non-blank symbolsoutput by a final Softmax layer. The prediction network is alsoconfigured to, for each non-blank symbol in the sequence of non-blanksymbols received as input at the corresponding time step: generate,using a shared embedding matrix, an embedding of the correspondingnon-blank symbol; assign a respective position vector to thecorresponding non-blank symbol; and weight the embedding proportional toa similarity between the embedding and the respective position vector.The prediction network is also configured to generate, as output, asingle embedding vector at the corresponding time step, the singleembedding vector based on a weighted average of the weighted embeddings.The RNN-T model also includes a joint network configured to, at each ofthe plurality of time steps subsequent to the initial time step:receive, as input, the single embedding vector generated as output fromthe prediction network at the corresponding time step; and generate aprobability distribution over possible speech recognition hypotheses atthe corresponding time step.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, the RNN-T modelfurther includes an audio encoder configured to receive, as input, asequence of acoustic frames and generate, at each of the plurality oftime steps, a higher order feature representation for a correspondingacoustic frame in the sequence of acoustic frames. Here, the jointnetwork is further configured to, at each of the plurality of timesteps, receive the higher order feature representation generated by theaudio encoder at the corresponding time step as input. In some examples,weighting the embedding proportional to the similarity between theembedding and the respective position vector includes weighting theembedding proportional to a cosine similarity between the embedding andthe respective position vector. The sequence of non-blank symbols outputby the final Softmax layer include wordpieces. Optionally, the sequenceof non-blank symbols output by the final

Softmax layer may include graphemes. Each of the embeddings may includea same dimension size as each of the position vectors. In someimplementations, the sequence of non-blank symbols received as input islimited to the N previous non-blank symbols output by the final Softmaxlayer. In these implementations, N may be equal to two. Alternatively, Nmay be equal to five.

In some examples, the prediction network includes a multi-headedattention mechanism that shares the shared embedding matrix across eachhead of the multi-headed attention mechanism. In these examples, at eachof the plurality of time steps subsequent to the initial time step theprediction network is configured to, at each head of the multi-headedattention mechanism, and for each non-blank symbol in the sequence ofnon-blank symbols received as input at the corresponding time step:generate, using the shared embedding matrix, the same embedding of thecorresponding non-blank symbol as the embedding generated at each otherhead of the multi-headed attention mechanism; assign a differentrespective position vector to the corresponding non-blank symbol thanthe respective position vector assigned to the corresponding non-blanksymbol at each other head of the multi-headed attention mechanism; andweight the embedding proportional to the similarity between theembedding and the respective position vector. Here, the predictionnetwork is also configured to generate, as output from the correspondinghead of the multi-headed attention mechanism, a respective weightedaverage of the weighted embeddings of the sequence of non-blank symbolsand generate, as output, the single embedding vector at thecorresponding time step by averaging the respective weighted averagesoutput from the corresponding heads of the multi-headed attentionmechanisms. In these examples, the multi-headed attention mechanism mayinclude four heads. Optionally, the prediction network may tie adimensionality of the shared embedding matrix to a dimensionality of anoutput layer of the joint network.

Another aspect of the disclosure provides a computer-implemented methodthat when executed on data processing hardware causes the dataprocessing hardware to perform operations. At each of a plurality oftime steps subsequent to an initial time step, the operations include:receiving, as input to a recurrent neural network-transducer (RNN-T)model, a sequence of non-blank symbols output by a final Softmax layer;for each non-blank symbol in the sequence of non-blank symbols receivedas input at the corresponding time step: generating, by the predictionnetwork, using a shared embedding matrix, and embedding of thecorresponding non-blank symbol; assigning, by the prediction network, arespective position vector to the corresponding non-blank symbol; andweighting , by the prediction network, the embedding proportional to asimilarity between the embedding and the respective position vector. Ateach of the plurality of time steps subsequent to the initial time step,the operations also include: generating, as output from the predictionnetwork, a single embedding vector at the corresponding time step; andgenerating, by a joint network of the RNN-T model, using the singleembedding vector generated as output from the prediction network at thecorresponding time step, a probability distribution over possible speechrecognition hypotheses at the corresponding time step. The singleembedding vector is based on a weighted average of the weightedembeddings.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, the operationsfurther include: receiving, as input to an audio encoder, a sequence ofacoustic frames; generating, by the audio encoder, at each of theplurality of time steps, a higher order feature representation for acorresponding acoustic frame in the sequence of acoustic frames; andreceiving, as input to the joint network, the higher order featurerepresentation generated by the audio encoder at the corresponding timestep. In some examples, weighting the embedding proportional to thesimilarity between the embedding and the respective position vectorincludes weighting the embedding proportional to a cosine similaritybetween the embedding and the respective position vector.

The sequence of non-blank symbols output by the final Softmax layer mayinclude wordpieces. Optionally, the sequence of non-blank symbols outputby the final

Softmax layer may include graphemes. Each of the embeddings may includea same dimension size as each of the position vectors. In someimplementations, the sequence of non-blank symbols received as input islimited to the N previous non-blank symbols output by the final Softmaxlayer. In these implementations, N may be equal to two. Alternatively, Nmay be equal to five. Optionally, the prediction network may tie adimensionality of the shared embedding matrix to a dimensionality of anoutput layer of the joint network.

In some examples, the prediction network includes a multi-headedattention mechanism that shares the shared embedding matrix across eachhead of the multi-headed attention mechanism. The multi-headed attentionmechanism may include four heads. In these examples, at each of theplurality of time steps subsequent to the initial time step, theoperations may further include, at each head of the multi-headedattention mechanism, and for each non-blank symbol in the sequence ofnon-blank symbols received as input at the corresponding time step:generating, by the prediction network and using the shared embeddingmatrix, the same embedding of the corresponding non-blank symbol as theembedding generated at each other head of the multi-headed attentionmechanism; assigning, by the prediction network, a different respectiveposition vector to the corresponding non-blank symbol than therespective position vector assigned to the corresponding non-blanksymbol at each other head of the multi-headed attention mechanism; andweighting, by the prediction network, the embedding proportional to thesimilarity between the embedding and the respective position vector.Here, at each of the plurality of time steps subsequent to the initialtime step and at each head of the multi-headed attention mechanism, theoperations may also include generating, by the prediction network asoutput from the corresponding head of the multi-headed attentionmechanism, a respective weighted average of the weighted embeddings ofthe sequence of non-blank symbols. Thereafter, at each of the pluralityof time steps subsequent to the initial time step, the operations mayfurther include generating, as output from the prediction network, thesingle embedding vector at the corresponding time step by averaging therespective weighted averages output from the corresponding heads of themulti-headed attention mechanisms.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of an example speech environment using arecurrent neural network-transducer (RNN-T) model for transcribingspeech.

FIG. 2 is a schematic view of an example RNN-T model architecture.

FIG. 3 is a schematic view of an example tied and reduced predictionnetwork of the RNN-T model architecture of FIG. 2.

FIG. 4 is a plot depicting word error rate versus size of both tied andun-tied prediction and joint networks.

FIG. 5 is a flowchart of an example arrangement of operations for acomputer-implemented method of executing a tied and reduced RNN-T model.

FIG. 6 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is an example of a speech environment 100. In the speechenvironment 100, a user's 104 manner of interacting with a computingdevice, such as a user device 10, may be through voice input. The userdevice 10 (also referred to generally as a device 10) is configured tocapture sounds (e.g., streaming audio data) from one or more users 104within the speech environment 100. Here, the streaming audio data mayrefer to a spoken utterance 106 by the user 104 that functions as anaudible query, a command for the device 10, or an audible communicationcaptured by the device 10. Speech-enabled systems of the device 10 mayfield the query or the command by answering the query and/or causing thecommand to be performed/fulfilled by one or more downstreamapplications.

The user device 10 may correspond to any computing device associatedwith a user 104 and capable of receiving audio data. Some examples ofuser devices 10 include, but are not limited to, mobile devices (e.g.,mobile phones, tablets, laptops, etc.), computers, wearable devices(e.g., smart watches), smart appliances, internet of things (IoT)devices, vehicle infotainment systems, smart displays, smart speakers,etc. The user device 10 includes data processing hardware 12 and memoryhardware 14 in communication with the data processing hardware 12 andstores instructions, that when executed by the data processing hardware12, cause the data processing hardware 12 to perform one or moreoperations. The user device 10 further includes an audio system 16 withan audio capture device (e.g., microphone) 16, 16 a for capturing andconverting spoken utterances 106 within the speech environment 100 intoelectrical signals and a speech output device (e.g., a speaker) 16, 16 bfor communicating an audible audio signal (e.g., as output audio datafrom the device 10). While the user device 10 implements a single audiocapture device 16 a in the example shown, the user device 10 mayimplement an array of audio capture devices 16 a without departing fromthe scope of the present disclosure, whereby one or more capture devices16 a in the array may not physically reside on the user device 10, butbe in communication with the audio system 16.

In the speech environment 100, an automated speech recognition (ASR)system 118 implementing a recurrent neural network-transducer (RNN-T)model 200 and an optional rescorer 180 resides on the user device 10 ofthe user 104 and/or on a remote computing device 60 (e.g., one or moreremote servers of a distributed system executing in a cloud-computingenvironment) in communication with the user device 10 via a network 40.The user device 10 and/or the remote computing device 60 also includesan audio subsystem 108 configured to receive the utterance 106 spoken bythe user 104 and captured by the audio capture device 16 a, and convertthe utterance 106 into a corresponding digital format associated withinput acoustic frames 110 capable of being processed by the ASR system118. In the example shown, the user speaks a respective utterance 106and the audio subsystem 108 converts the utterance 106 intocorresponding audio data (e.g., acoustic frames) 110 for input to theASR system 118. Thereafter, the RNN-T model 200 receives, as input, theaudio data 110 corresponding to the utterance 106, andgenerates/predicts, as output, a corresponding transcription 120 (e.g.,recognition result/hypothesis) of the utterance 106. In the exampleshown, the RNN-T model 200 may perform streaming speech recognition toproduce an initial speech recognition result 120, 120 a and the rescorer180 may update (i.e., rescore) the initial speech recognition result 120a to produce a final speech recognition result 120, 120 b.

The user device 10 and/or the remote computing device 60 also executes auser interface generator 107 configured to present a representation ofthe transcription 120 of the utterance 106 to the user 104 of the userdevice 10. As described in greater detail below, the user interfacegenerator 107 may display the initial speech recognition results 120 ain a streaming fashion during time 1 and subsequently display the finalspeech recognition result 120 b during time 2. In some configurations,the transcription 120 output from the ASR system 118 is processed, e.g.,by a natural language understanding (NLU) module executing on the userdevice 10 or the remote computing device 60, to execute a usercommand/query specified by the utterance 106. Additionally oralternatively, a text-to-speech system (not shown) (e.g., executing onany combination of the user device 10 or the remote computing device 60)may convert the transcription into synthesized speech for audible outputby the user device 10 and/or another device.

In the example shown, the user 104 interacts with a program orapplication 50 (e.g., the digital assistant application 50) of the userdevice 10 that uses the ASR system 118. For instance, FIG. 1 depicts theuser 104 communicating with the digital assistant application 50 and thedigital assistant application 50 displaying a digital assistantinterface 18 on a screen of the user device 10 to depict a conversationbetween the user 104 and the digital assistant application 50. In thisexample, the user 104 asks the digital assistant application 50, “Whattime is the concert tonight?” This question from the user 104 is aspoken utterance 106 captured by the audio capture device 16 a andprocessed by audio systems 16 of the user device 10. In this example,the audio system 16 receives the spoken utterance 106 and converts itinto acoustic frames 110 for input to the ASR system 118.

Continuing with the example, the RNN-T model 200, while receiving theacoustic frames 110 corresponding to the utterance 106 as the user 104speaks, encodes the acoustic frames 110 and then decodes the encodedacoustic frames 110 into the initial speech recognition results 120 a.During time 1, the user interface generator 107 presents, via thedigital assistant interface 18, a representation of the initial speechrecognition results 120 a of the utterance 106 to the user 104 of theuser device 10 in a streaming fashion such that words, word pieces,and/or individual characters appear on the screen as soon as they arespoken. In some examples, the first look ahead audio context is equal tozero.

During time 2, the user interface generator 107 presents, via thedigital assistant interface 18, a representation of the final speechrecognition result 120 b of the utterance 106 to the user 104 of theuser device 10 rescored by the rescorer 180. In some implementations,the user interface generator 107 replaces the representation of theinitial speech recognition results 120 a presented at time 1 with therepresentation of the final speech recognition result 120 b presented attime 2. Here, time 1 and time 2 may include timestamps corresponding towhen the user interface generator 107 presents the respective speechrecognition result 120. In this example, the timestamp of time 1indicates that the user interface generator 107 presents the initialspeech recognition result 120 a at an earlier time than the final speechrecognition result 120 b. For instance, as the final speech recognitionresult 120 b is presumed to be more accurate than the initial speechrecognition results 120 a, the final speech recognition result 120 bultimately displayed as the transcription 120 may fix any terms that mayhave been misrecognized in the initial speech recognition results 120 a.In this example, the streaming initial speech recognition results 120 aoutput by the RNN-T model 200 are displayed on the screen of the userdevice 10 at time 1 are associated with low latency and provideresponsiveness to the user 104 that his/her query is being processed,while the final speech recognition result 120 b output by the rescorer180 and displayed on the screen at time 2 leverages an additional speechrecognition model and/or a language model to improve the speechrecognition quality in terms of accuracy, but at increased latency.However, since the initial speech recognition results 120 a aredisplayed as the user speaks the utterance 106, the higher latencyassociated with producing, and ultimately displaying the finalrecognition result is not noticeable to the user 104.

In the example shown in FIG. 1, the digital assistant application 50 mayrespond to the question posed by the user 104 using natural languageprocessing. Natural language processing generally refers to a process ofinterpreting written language (e.g., the initial speech recognitionresults 120 a and/or the final speech recognition result 120 b) anddetermining whether the written language prompts any action. In thisexample, the digital assistant application 50 uses natural languageprocessing to recognize that the question from the user 104 regards theuser's schedule and more particularly a concert on the user's schedule.By recognizing these details with natural language processing, theautomated assistant returns a response 19 to the user's query where theresponse 19 states, “Venue doors open at 6:30 PM and concert starts at 8pm.” In some configurations, natural language processing occurs on aremote server 60 in communication with the data processing hardware 12of the user device 10.

Referring to FIG. 2, an example frame alignment-based transducer model200 includes a Recurrent Neural Network-Transducer (RNN-T) modelarchitecture which adheres to latency constraints associated withinteractive applications. The RNN-T model 200 provides a smallcomputational footprint and utilizes less memory requirements thanconventional ASR architectures, making the RNN-T model architecturesuitable for performing speech recognition entirely on the user device102 (e.g., no communication with a remote server is required). The RNN-Tmodel 200 includes an encoder network 210, a prediction network 300, anda joint network 230.

The prediction and joint networks 300, 230 may collectively provide anRNN-T decoder. The encoder network 210, which is roughly analogous to anacoustic model (AM) in a traditional ASR system, includes a recurrentnetwork of stacked Long Short-Term Memory (LSTM) layers. For instance,the encoder reads a sequence of d-dimensional feature vectors (e.g.,acoustic frames 110 (FIG. 1)) x=(x₁, x₂, . . . , x_(T)), where x_(t) ∈

_(d), and produces at each time step a higher-order featurerepresentation. This higher-order feature representation is denoted ash₁ ^(enc), . . , h_(T) ^(enc).

Similarly, the prediction network 300 is also an LSTM network, which,like a language model (LM), processes the sequence of non-blank symbols301 output by a final Softmax layer 240 so far, y₀, . . . , y_(ui-1),into a representation P_(u) _(i) . Described in greater detail below,the representation P_(u) _(i) 350 includes a single embedding vector.Notably, the sequence of non-blank symbols 301 received at theprediction network 300 capture linguistic dependencies between non-blanksymbols 301 predicted during the previous time steps so far to assistthe joint network 230 in predicting the probability of a next outputsymbol or blank symbol during the current time step. As described ingreater detail below, to contribute to techniques for reducing the sizeof the prediction network 300 without sacrificing accuracy/performanceof the RNN-T model 200, the prediction network 300 may receive alimited-history sequence of non-blank symbols y_(ui-n), . . . , y_(ui-1)that is limited to the N previous non-blank symbols 301 output by thefinal Softmax layer 240.

Finally, with the RNN-T model architecture, the representations producedby the encoder and prediction networks 210, 300 are combined by thejoint network 230. The joint network then predicts Z_(i)=P(y_(i)|x_(t)_(i) , y₀, . . . , y_(u) _(i−1) ), which is a distribution over the nextoutput symbol. Stated differently, the joint network 230 generates, ateach output step (e.g., time step), a probability distribution overpossible speech recognition hypotheses. Here, the “possible speechrecognition hypotheses” correspond to a set of output labels eachrepresenting a symbol/character in a specified natural language. Forexample, when the natural language is English, the set of output labelsmay include twenty-seven (27) symbols, e.g., one label for each of the26-letters in the English alphabet and one label designating a space.Accordingly, the joint network 230 may output a set of values indicativeof the likelihood of occurrence of each of a predetermined set of outputlabels. This set of values can be a vector and can indicate aprobability distribution over the set of output labels. In some cases,the output labels are graphemes (e.g., individual characters, andpotentially punctuation and other symbols), but the set of output labelsis not so limited. For example, the set of output labels can includewordpieces and/or entire words, in addition to or instead of graphemes.The output distribution of the joint network 230 can include a posteriorprobability value for each of the different output labels. Thus, ifthere are 100 different output labels representing different graphemesor other symbols, the output yi of the joint network 230 can include 100different probability values, one for each output label. The probabilitydistribution can then be used to select and assign scores to candidateorthographic elements (e.g., graphemes, wordpieces, and/or words) in abeam search process (e.g., by the Softmax layer 240) for determining thetranscription 120.

The Softmax layer 240 may employ any technique to select the outputlabel/symbol with the highest probability in the distribution as thenext output symbol predicted by the RNN-T model 200 at the correspondingoutput step. In this manner, the RNN-T model 200 does not make aconditional independence assumption, rather the prediction of eachsymbol is conditioned not only on the acoustics but also on the sequenceof labels output so far. The RNN-T model 200 does assume an outputsymbol is independent of future acoustic frames 110, which allows theRNN-T model to be employed in a streaming fashion.

In some examples, the encoder network 210 of the RNN-T model 200 is madeup of eight 2,048-dimensional LSTM layers, each followed by a640-dimensional projection layer. In other implementations, the encodernetwork 210 includes a network of conformer or transformer layers. Theprediction network 220 may have two 2,048-dimensional LSTM layers, eachof which is also followed by 640-dimensional projection layer as well asan embedding layer of 128 units. Finally, the joint network 230 may alsohave 640 hidden units. The Softmax layer 240 may be composed of aunified word piece or grapheme set that is generated using all uniqueword pieces or graphemes in training data. When the outputsymbols/labels include wordpieces, the set of output symbols/labels mayinclude 4,096 different word pieces. When the output symbols/labelsinclude graphemes, the set of output symbols/labels may include lessthan 100 different graphemes.

FIG. 3 shows the prediction network 300 of the RNN-T model 200receiving, as input, a sequence of non-blank symbols y_(ui-n). . . ,y_(ui-1) that is limited to the N previous non-blank symbols 301 a-noutput by the final Softmax layer 240. In some examples, N is equal totwo. In other examples, N is equal to five, however, the disclosure isnon-limiting and N may equal any integer. The sequence of non-blanksymbols 301 a-n indicates an initial speech recognition result 120 a(FIG. 1). In some implementations, the prediction network 300 includes amulti-headed attention mechanism 302 that shares a shared embeddingmatrix 304 across each head 302A-302H of the multi-headed attentionmechanism. In one example, the multi-headed attention mechanism 302includes four heads. However, any number of heads may be employed by themulti-headed attention mechanism 302. Notably, the multi-headedattention mechanism improves performance significantly with minimalincrease to model size. As described in greater detail below, each head302A-H includes its own row of position vectors 308, and rather thanincurring an increase in model size by concatenating outputs 318A-H fromall the heads, the outputs 318A-H are instead averaged by a head averagemodule 322.

Referring to the first head 302A of the multi-headed attention mechanism302, the head 302A generates, using the shared embedding matrix 304, acorresponding embedding 306, 306 a-n (e.g., X ∈

^(N×d) ^(e) ) for each non-blank symbol 301 among the sequence ofnon-blank symbols y_(ui-n). . . , y_(ui-1) received as input at thecorresponding time step from the plurality of time steps. Notably, sincethe shared embedding matrix 304 is shared across all heads of themulti-headed attention mechanism 302, the other heads 302B-H allgenerate the same corresponding embeddings 306 for each non-blanksymbol. The head 302A also assigns a respective position vectorPV_(Aa-An) 308, 308Aa-An (e.g., P ∈

^(H×N×d) ^(e) ) to each corresponding non-blank symbol in the sequenceof non-blank symbols y_(ui-n), . . . , y_(ui-1). The respective positionvector PV 308 assigned to each non-blank symbol indicates a position inthe history of the sequence of non-blank symbols (e.g., the N previousnon-blank symbols output by the final Softmax layer 240). For instance,the first position vector PV_(Aa) is assigned to a most recent positionin the history, while the last position vector PV_(An) is assigned to alast position in the history of the N previous non-blank symbols outputby the final Softmax layer 240. Notably, each of the embeddings 306 mayinclude a same dimensionality (i.e., dimension size) as each of theposition vectors PV 308.

While the corresponding embedding generated by shared embedding matrix304 for each for each non-blank symbol 301 among the sequence ofnon-blank symbols 301 a-n, . . . , y_(ui-1), is the same at all of theheads 302A-H of the multi-headed attention mechanism 302, each head302A-H defines a different set/row of position vectors 308. Forinstance, the first head 302A defines the row of position vectorsPV_(Aa-An) 08Aa-An, the second head 302B defines a different row ofposition vectors PVBa-Bn 308 _(Ba-Bn), . . . , and the Hth head 302 Hdefines another different row of position vectors PVHa-Hn 308Ha-Hn.

For each non-blank symbol in the sequence of non-blank symbols 301 a-nreceived, the first head 302A also weights, via a weight layer 310, thecorresponding embedding 306 proportional to a similarity between thecorresponding embedding and the respective position vector PV 308assigned thereto. In some examples, the similarity may include a cosinesimilarity (e.g., cosine distance). In the example shown, the weightlayer 310 outputs a sequence of weighted embeddings 312, 312Aa-An eachassociated the corresponding embedding 306 weighted proportional to therespective position vector PV 308 assigned thereto. Stated differently,the weighted embeddings 312 output by the weight layer 310 for eachembedding 306 may correspond to a dot product between the embedding 306and the respective position vector PV 308. The weighted embeddings 312may be interpreted as attending over the embeddings in proportion to howsimilar they are to the positioned associated with their respectiveposition vectors PV 308. To increase computational speed, the predictionnetwork 300 includes non-recurrent layers, and therefore, the sequenceof weighted embeddings 312Aa-An are not concatenated, but instead,averaged by a weighted average module 316 to generate, as output fromthe first head 302A, a weighted average 318A of the weighted embeddings312Aa-An represented by:

$\begin{matrix}{{{Prediction}\left( {X,P} \right)} = {\frac{1}{H*N}{\sum\limits_{h,n}{X_{n}*{\sum\limits_{e}\left( {X_{n,e}*P_{h,n,e}} \right)}}}}} & (1)\end{matrix}$

In Equation 1, h represents the index of the heads 302, n representsposition in context, and e represents the embedding dimension.Additionally, in Equation 1, H, N, and de include the sizes of thecorresponding dimensions. The position vector PV 308 does not have to betrainable and may include random values. Notably, even though theweighted embeddings 312 are averaged, the position vectors PV 308 canpotentially save position history information, alleviating the need toprovide recurrent connections at each layer of the prediction network300.

The operations described above with respect to the first head 302A, aresimilarly performed by each other head 302B-H of the multi-headedattention mechanism 302. Due to the different set of positioned vectorsPV 308 defined by each head 302, the weight layer 310 outputs a sequenceof weighted embeddings 312Ba-Bn, 312Ha-Hn at each other head 302B-H thatis different than the sequence of weighted embeddings 312Aa-Aa at thefirst head 302A. Thereafter, the weighted average module 316 generates,as output from each other corresponding head 302B-H, a respectiveweighted average 318B-H of the corresponding weighted embeddings 312 ofthe sequence of non-blank symbols.

In the example shown, the prediction network 300 includes a head averagemodule 322 that averages the weighted averages 318A-H output from thecorresponding heads 302A-H. A projection layer 326 with SWISH mayreceive, as input, an output 324 from the head average module 322 thatcorresponds to the average of the weighted averages 318A-H, andgenerate, as output, a projected output 328. A final layer normalization330 may normalize the projected output 328 to provide the singleembedding vector Pu_(i) 350 at the corresponding time step from theplurality of time steps. The prediction network 300 generates only asingle embedding vector Pu_(i) 350 at each of the plurality of timesteps subsequent to an initial time step.

In some configurations, the prediction network 300 does not implementthe multi-headed attention mechanism 302 and only performs theoperations described above with respect to the first head 302A. In theseconfigurations, the weighted average 318A of the weighted embeddings312Aa-An is simply passed through the projection layer 326 and layernormalization 330 to provide the single embedding vector Pu_(i) 350.

Referring back to FIG. 2, the joint network 230 receives the singleembedding vector Pu_(i) 350 from the prediction network 300 and thehigher-order feature representation h_(t) _(i) ^(enc) from the encoder210. The joint network 230 generates a probability distributionP(y_(i)|x_(t) _(i) , y₀, . . . , y_(u) _(t-i) ) over possible speechrecognition hypotheses at the corresponding time step. Here, thepossible speech recognition hypotheses correspond to a set of outputlabel that each represent a symbol character in a specified naturallanguage. The probability distribution P(y_(i)|x_(t) _(i) , y₀, . . . ,y_(u) _(u-i) ) over the possible speech recognition hypotheses indicatesa probability for the final speech recognition result 120 b (FIG. 1).That is, the joint network 230 determines the probability distributionfor the final speech recognition result 120 b using the single embeddingvector 350 that is based on the sequence of non-blank symbols (e.g.,initial speech recognition result 120 a). The final Softmax layer 240receives the probability distribution for the final speech recognitionresult 120 b and selects the output label/symbol with the highestprobability to produce the transcription.

The final speech recognition result 120 b is presumed to be moreaccurate than the initial speech recognition result 120 a because theRNN-T model 200 determines the initial speech recognition results 120 ain a streaming fashion and the final speech recognition results 120 busing the prior non-blank symbols from the initial speech recognitionresult 120 a. That is, the final speech recognition results 120 b takeinto account the prior non-blank symbols and thus are presumed moreaccurate because the initial speech recognition results 120 a do nottake into account any prior non-blank symbols. Moreover, the rescorer180 (FIG. 1) may update the initial speech recognition result 120 a withthe final speech recognition result 120 b to provide the transcriptionvia the user interface generator 170 to the user 104.

In some implementations, to further reduce the size of the RNN-Tdecoder, i.e., the prediction network 300 and the joint network 230,parameter tying between the prediction network 300 and the joint network230 is applied. Specifically, for a vocabulary size |V| and an embeddingdimension de, the shared embedding matrix 304 at the prediction networkis E ∈

^(|V|×d) ^(e) . Meanwhile, a last hidden layer includes a dimension sized_(h) at the joint network 230, feed-forward projection weights from thehidden layer to the output logits will be W ∈

^(d) ^(h) ^(×|V+1|), with an extra blank token in the vocabulary.Accordingly, the feed-forward layer corresponding to the last layer ofthe joint network 230 includes a weight matrix [d_(h), |V]|. By havingthe prediction network 300 to tie the size of the embedding dimension deto the dimensionality d_(h) of the last hidden layer of the jointnetwork 230, the feed-forward projection weights of the joint network230 and the shared embedding matrix 304 of the prediction network 300can share their weights for all non-blank symbols via a simple transposetransformation. Since the two matrices share all their values, the RNN-Tdecoder only needs to store the values once on memory, instead ofstoring two individual matrices. By setting the size of the embeddingdimension de equal to the size of the hidden layer dimension dh, theRNN-T decoder reduces a number of parameters equal to the product of theembedding dimension de and the vocabulary size V. This weight tyingcorresponds to a regularization technique.

FIG. 4 is a plot 400 depicting word error rate (WER) versus the numberof parameters of the RNN-T decoders. Here, FIG. 4 the plot 400illustrates WER versus the number of parameters for a tied RNN-T decoder410 (illustrated with the solid line), a non-tied RNN-T decoder 420(illustrated with the dotted line), and a long-short term memory (LSTM)network 430 (illustrated with the dashed line). Specifically, the plot400 depicts the size of the prediction and joint networks 300, 230 withand without tied output and embeddings. The plot 400 shows varyingembedding dimensions de to perform a sweep over the model size. As shownin FIG. 4, the non-tied RNN-T decoder 420 includes four measurementsincluding the embedding dimensions de of 64, 320, 640, 960, and 1280.Here, the tied RNN-T decoder 410 includes three measurements includingthe embedding dimensions of 640, 960, and 1280. In the non-tied RNN-Tdecoder 420 case, the last hidden layer of the joint network 230 alwaysincludes the dimensionality d_(h) of size 640 (dimensionality d_(h) notillustrated in the plot 400). In the tied RNN-T decoder 410 case, theplot 400 also shows the dimensionality d_(h) (dimensionality d_(h) notillustrated in the plot 400) of the last hidden layer of the jointnetwork 230 equal to the size of the embedding dimension de of theprediction network 300 such that the size and performance of the RNN-Tdecoder is more sensitive to changes in that dimension. Accordingly, theresults depicted by plot 400 indicate that weight-tying is moreparameter efficient, thereby achieving better performance with fewerparameters. Additionally, for large-enough models using weight-tying,the same word error rate is reached as a conventional RNN-T decoderusing the LSTM network 430.

FIG. 5 is a flowchart of an exemplary arrangement of operations for acomputer-implemented method 500 for executing a tied and reduced RNN-Tmodel 200. At each of a plurality of time steps subsequent to an initialtime step, the method 500 performs operations 502-512. At operation 502,the method 500 includes receiving, as input to a prediction network 300of a recurrent neural network-transducer (RNN-T) model 200, a sequenceof non-blank symbols 301, 301 a-ny_(ui-n), . . . , y_(ui-1) output by afinal Softmax layer 240. For each non-blank symbol in the sequence ofnon-blank symbols received as input during the corresponding time step,the method 500 performs operations 504-508. At operation 504, the method500 includes generating, by the prediction network 300, using a sharedembedding matrix 304, an embedding 306 of the corresponding non-blanksymbol. At operation 506, the method 500 includes assigning, by theprediction network 300, a respective position vector PV_(Aa-An) 308,308Aa-An to the corresponding non-blank symbol. At operation 508, themethod 500 includes weighting, by the prediction network 300, theembedding 306 proportional to a similarity between the embedding 306 andthe respective position vector 308.

At operation 510, the method 500 includes generating, as output from theprediction network 300, a single embedding vector 350 at thecorresponding time step. Here, the single embedding vector 350 is basedon a weighted average 318A-H of the weighted embeddings 312Aa-An. Atoperation 512, the method 500 includes generating, by a joint network230 of the RNN-T model 200, using the single embedding vector 350generated as output from the prediction network 300 at the correspondingtime step, a probability distribution P(y_(i)|x_(t) _(i) , y₀, . . . ,y_(u) _(i-1) ) over possible speech recognition hypotheses at thecorresponding time step.

FIG. 6 is schematic view of an example computing device 600 that may beused to implement the systems and methods described in this document.The computing device 600 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

The computing device 600 includes a processor 610, memory 620, a storagedevice 630, a high-speed interface/controller 640 connecting to thememory 620 and high-speed expansion ports 650, and a low speedinterface/controller 660 connecting to a low speed bus 670 and a storagedevice 630. Each of the components 610, 620, 630, 640, 650, and 660, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 610 canprocess instructions for execution within the computing device 600,including instructions stored in the memory 620 or on the storage device630 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 680 coupled tohigh speed interface 640. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 600 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 620 stores information non-transitorily within the computingdevice 600. The memory 620 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 620 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 600.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 630 is capable of providing mass storage for thecomputing device 600. In some implementations, the storage device 630 isa computer-readable medium. In various different implementations, thestorage device 630 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 620, the storage device 630,or memory on processor 610.

The high speed controller 640 manages bandwidth-intensive operations forthe computing device 600, while the low speed controller 660 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 640is coupled to the memory 620, the display 680 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 650,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 660 is coupled to the storagedevice 630 and a low-speed expansion port 690. The low-speed expansionport 690, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 600 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 600 a or multiple times in a group of such servers 600a, as a laptop computer 600 b, or as part of a rack server system 600 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

A software application (i.e., a software resource) may refer to computersoftware that causes a computing device to perform a task. In someexamples, a software application may be referred to as an “application,”an “app,” or a “program.” Example applications include, but are notlimited to, system diagnostic applications, system managementapplications, system maintenance applications, word processingapplications, spreadsheet applications, messaging applications, mediastreaming applications, social networking applications, and gamingapplications.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A recurrent neural network-transducer (RNN-T)model comprising: a prediction network configured to, at each of aplurality of time steps subsequent to an initial time step: receive, asinput, a sequence of non-blank symbols output by a final Softmax layer;for each non-blank symbol in the sequence of non-blank symbols receivedas input at the corresponding time step: generate, using a sharedembedding matrix, an embedding of the corresponding non-blank symbol;assign a respective position vector to the corresponding non-blanksymbol; and weight the embedding proportional to a similarity betweenthe embedding and the respective position vector; and generate, asoutput, a single embedding vector at the corresponding time step, thesingle embedding vector based on a weighted average of the weightedembeddings; and a joint network configured to, at each of the pluralityof time steps subsequent to the initial time step: receive, as input,the single embedding vector generated as output from the predictionnetwork at the corresponding time step; and generate a probabilitydistribution over possible speech recognition hypotheses at thecorresponding time step.
 2. The RNN-T model of claim 1, furthercomprising: an audio encoder configured to: receive, as input, asequence of acoustic frames; and generate, at each of the plurality oftime steps, a higher order feature representation for a correspondingacoustic frame in the sequence of acoustic frames, wherein the jointnetwork is further configured to, at each of the plurality of timesteps, receive the higher order feature representation generated by theaudio encoder at the corresponding time step as input.
 3. The RNN-Tmodel of claim 1, wherein weighting the embedding proportional to thesimilarity between the embedding and the respective position vectorcomprises weighting the embedding proportional to a cosine similaritybetween the embedding and the respective position vector.
 4. The RNN-Tmodel of claim 1, wherein the sequence of non-blank symbols output bythe final Softmax layer comprise wordpieces.
 5. The RNN-T model of claim1, wherein the sequence of non-blank symbols output by the final Softmaxlayer comprise graphemes.
 6. The RNN-T model of claim 1, wherein each ofthe embeddings comprise a same dimension size as each of the positionvectors.
 7. The RNN-T model of claim 1, wherein the sequence ofnon-blank symbols received as input is limited to N previous non-blanksymbols output by the final Softmax layer.
 8. The RNN-T model of claim7, wherein N is equal to two.
 9. The RNN-T model of claim 7, wherein Nis equal to five.
 10. The RNN-T model of claim 1, wherein the predictionnetwork comprises a multi-headed attention mechanism, the multi-headedattention mechanism sharing the shared embedding matrix across each headof the multi-headed attention mechanism.
 11. The RNN-T model of claim10, wherein the prediction network is configured to, at each of theplurality of time steps subsequent to the initial time step: at eachhead of the multi-headed attention mechanism: for each non-blank symbolin the sequence of non-blank symbols received as input at thecorresponding time step: generate, using the shared embedding matrix,the same embedding of the corresponding non-blank symbol as theembedding generated at each other head of the multi-headed attentionmechanism; assign a different respective position vector to thecorresponding non-blank symbol than the respective position vectorsassigned to the corresponding non-blank symbol at each other head of themulti-headed attention mechanism; and weight the embedding proportionalto the similarity between the embedding and the respective positionvector; and generate, as output from the corresponding head of themulti-headed attention mechanism, a respective weighted average of theweighted embeddings of the sequence of non-blank symbols; and generate,as output, the single embedding vector at the corresponding time step byaveraging the respective weighted averages output from the correspondingheads of the multi-headed attention mechanism.
 12. The RNN-T model ofclaim 10, wherein the multi-headed attention mechanism comprises fourheads.
 13. The RNN-T model of claim 1, wherein the prediction networkties a dimensionality of the shared embedding matrix to a dimensionalityof an output layer of the joint network.
 14. A computer-implementedmethod when executed on data processing hardware causes the dataprocessing hardware to perform operations comprising: at each of aplurality of time steps subsequent to an initial time step: receiving,as input to a prediction network of a recurrent neuralnetwork-transducer (RNN-T) model, a sequence of non-blank symbols outputby a final Softmax layer; for each non-blank symbol in the sequence ofnon-blank symbols received as input at the corresponding time step:generating, by the prediction network, using a shared embedding matrix,an embedding of the corresponding non-blank symbol; assigning, by theprediction network, a respective position vector to the correspondingnon-blank symbol; and weighting, by the prediction network, theembedding proportional to a similarity between the embedding and therespective position vector; generating, as output from the predictionnetwork, a single embedding vector at the corresponding time step, thesingle embedding vector based on a weighted average of the weightedembeddings; and generating, by a joint network of the RNN-T model, usingthe single embedding vector generated as output from the predictionnetwork at the corresponding time step, a probability distribution overpossible speech recognition hypotheses at the corresponding time step.15. The computer-implemented method of claim 14, wherein the operationsfurther comprise: receiving, as input to an audio encoder of the RNN-Tmodel, a sequence of acoustic frames; generating, by the audio encoder,at each of the plurality of time steps, a higher order featurerepresentation for a corresponding acoustic frame in the sequence ofacoustic frames; and receiving, as input to the joint network, thehigher order feature representation generated by the audio encoder atthe corresponding time step.
 16. The computer-implemented method ofclaim 14, wherein weighting the embedding proportional to the similaritybetween the embedding and the respective position vector comprisesweighting the embedding proportional to a cosine similarity between theembedding and the respective position vector.
 17. Thecomputer-implemented method of claim 14, wherein the sequence ofnon-blank symbols output by the final Softmax layer comprise wordpieces.18. The computer-implemented method of claim 14, wherein the sequence ofnon-blank symbols output by the final Softmax layer comprise graphemes.19. The computer-implemented method of claim 14, wherein each of theembeddings comprise a same dimension size as each of the positionvectors.
 20. The computer-implemented method of claim 14, wherein thesequence of non-blank symbols received as input is limited to N previousnon-blank symbols output by the final Softmax layer.
 21. Thecomputer-implemented method of claim 20, wherein N is equal to two. 22.The computer-implemented method of claim 20, wherein N is equal to five.23. The computer-implemented method of claim 14, wherein the predictionnetwork comprises a multi-headed attention mechanism, the multi-headedattention mechanism sharing the shared embedding matrix across each headof the multi-headed attention mechanism.
 24. The computer implementedmethod of claim 23, wherein the operations further comprise, at each ofthe plurality of time steps subsequent to the initial time step: at eachhead of the multi-headed attention mechanism: for each non-blank symbolin the sequence of non-blank symbols received as input at thecorresponding time step: generating, by the prediction network, usingthe shared embedding matrix, the same embedding of the correspondingnon-blank symbol as the embedding generated at each other head of themulti-headed attention mechanism; assigning, by the prediction network,a different respective position vector to the corresponding non-blanksymbol than the respective position vectors assigned to thecorresponding non-blank symbol at each other head of the multi-headedattention mechanism; and weighting, by the prediction network, theembedding proportional to the similarity between the embedding and therespective position vector; and generating, by the prediction network,as output from the corresponding head of the multi-headed attentionmechanism, a respective weighted average of the weighted embeddings ofthe sequence of non-blank symbols; and generating, as output from theprediction network, the single embedding vector at the correspondingtime step by averaging the respective weighted averages output from thecorresponding heads of the multi-headed attention mechanism.
 25. Thecomputer-implemented method of claim 23, wherein the multi-headedattention mechanism comprises four heads.
 26. The computer-implementedmethod of claim 14, wherein the prediction network ties a dimensionalityof the shared embedding matrix to a dimensionality of an output layer ofthe joint network.